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PATENT 

Attorney Docket No.: 021 572-0002 10US 

ELECTRONIC MESSAGING SYNCHRONIZED TO MEDIA 

PRESENTATION 

CROSS-REFERENCES TO RELATED APPLICATIONS 
5 [01] This application claims priority to U.S. Provisional No. 60/402,358 filed August 8, 
2002, which is incorporated by reference herein. 

' BACKGROUND OF THE INVENTION 

1 0 [02] This invention relates in general to playback of an audiovisual production and 
more specifically to playback of an audiovisual production from a Digital Versatile Disk 
(DVD) while allowing the creation and display of synchronized text messages. 
[03] Playback of audiovisual productions using DVDs and DVD players is growing in 
popularity. Audiovisual productions include music videos, concerts, movies, games, public 

1 5 or private events and other similar types of recorded performances. As used herein, 

audiovisual productions also generally encompass audio productions, visual productions or 
other media presentations recorded on a compact disk (CD), digital versatile disk (DVD) or 
other such storage medium or transmitted over satellite or cable systems. Audiovisual 
productions can be viewed from computer systems, consumer digital devices, and other 

20 displays, systems or platforms. Consumer digital devices include video Compact Disk (CD) 
players or DVD players. Other platforms, such as MPEG players, allow storage or streaming 
of audiovisual productions over digital networks or the Internet. Unfortunately, playing or 
viewing previously recorded audiovisual productions are relatively static and do not change. 
Accordingly, there is diminished motivation for the viewer to return to the production over 

25 time. What is needed is a method to enhance the viewing experience and to provide the 
viewer greater insight into the production each time the production is viewed. 
[04] Exchanging text messaging over a digital network is also gaining popularity. 
Various communication techniques for text messages have been developed. For example, 
messaging systems based on "bulletin boards" usually provide a common set of functionality 

30 to enable open discussions that are available to all viewers in an organized and efficient 

manner. The organization and efficiency in communication are accomplished with the use of 
"threads," which are essentially subject hierarchies. Under each subject hierarchy, an initial 



message on any topic starts a "thread," and all messages in response to this initial message 
are organized with this "thread." All messages within the same thread are usually organized 
on a viewer interface with indentations, where the initial message of the thread appears in the 
leftmost position, and replies to this message are indented. Such indentations provide an 
5 easy-to-scan hierarchy of content. Further, text messaging systems based on "bulletin 

boards" may provide multiple levels of threading whereby responses to an initial message can 
initiate new, "deeper" threads of their own, thus creating a nested table of replies that can be 
at once freeform and highly organized. 

[05] However, what is needed is a method for viewers of audiovisual productions to 
1 0 discuss the production with other viewers in a manner that encourages communication and 
that enhances the exchange of opinions and information. Accordingly, there is a great need 
for coupling features of message threads with the viewing of previously recorded audiovisual 
productions. However, because many comments are scene specific, the threads need to be 
linked to specific frames or groups of frames of the audiovisual production. Further, what is 
15 needed is a messaging system that provides communications of messages over digital 
networks that are linked to the performance of an audiovisual production to improve a 
viewer's enjoyment and inform the viewer upon playback of an audiovisual production. 

20 BRIEF SUMMARY OF THE INVENTION 

[06] The present invention relates to a method for enhancing the viewing of static or 
previously recorded audiovisual productions by linking a messaging system to the viewing 
process to allow viewers to post messages, and replies to those messages, on threaded 

25 "bulletin boards" relating to each audiovisual production with each message linked to the 
audio and/or video media timeline. During the viewing process, messages appear at certain 
moments, and exist in a messaging interface region for a specific duration - allowing 
messages to be linked to specific moments in the audio/video presentation. The messages are 
threaded so that viewers can reply to messages in an organized fashion - allowing a 

30 community of viewers to discuss the media content asynchronously, at leisure, but also with a 
sense of being "in the moment" as the discussion is tied to their own current experience of 
audiovisual production. The representation of messages is such that any viewer can easily 
find the information they want to discuss as the method provides simple components to 
respond to post and to filter out unwanted information. 
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[07] The messaging system provides the tools to link a web-based threaded messaging 
system to media playback time code. In this manner, artists and related creative/editorial 
personnel, as well as regular viewers, can create messages presented with audiovisual 
production that are tied to moments in the media available on the discs. The message threads 
5 are stored on a server accessible via the Internet, so that any computer connected to the 
communication network will be able to pull messages into a messaging client interface. 
Thus, the present invention creates another avenue of channeling new, fresh content to an 
otherwise static interactive experience. 

[08] It will be appreciated that the amount of viewing space is limited and the viewing 
1 0 window is limited by the play rate. Thus, a key challenge is to simplify the messaging 

functionality so that it fits within the limited screen real estate given a limited user control 
interface, so that the presentation of the message threads do not significantly interfere with 
the viewer's viewing experience. Further, due to the limited space and time, filtering of less 
relevant threads, or portions of the threads, are provided so that viewers can restrict the 
1 5 displayed threads to only those postings and/or authors that they wish to see (again, within a 
limited control interface). When the viewer does not wish to see to see the messages, the 
audiovisual production may be displayed without messages. In this manner, the audiovisual 
production uses the entire screen real estate. When the viewer comes to an interesting portion 
of the audiovisual production, the viewer activates the messaging system and views 
20 messages. When finished viewing the messages, the viewer may switch back to the full 
screen audiovisual production. 

[09] In accordance with the present invention, a method for providing electronic 
messaging synchronized to an audiovisual production is provided. While streaming an 
audiovisual production to a display device, a viewer may simultaneously view message 

25 threads relating to the production. The method includes receiving a signal defining a 

message, which may include alphanumeric text, audio or pictures, from a viewer interface, 
displaying the defined message adjacent to a display of the audiovisual production. 
[10] In another embodiment of the present invention, a method for synchronizing 
messages input by the viewer with playback of an audiovisual production includes 

30 associating a given message thread with a determined point in time of the playback, and 
displaying the given message at the determined point in time during the playback. 
[11] In yet another embodiment of the present invention, a method for displaying text in 
connection with playback of an audiovisual production includes providing playback of an 
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audiovisual production on a display platform, and displaying a message thread input by the 
viewer regardless of the display platform or point in time of the playback, 
f 12] In yet another embodiment of the present invention, a system for displaying a 
message thread in connection with playback of prerecorded media to produce an audiovisual 
5 production that includes a receiving system for receiving a signal from a viewer interface to 
define a message containing alphanumeric text, images or sound during playback of the 
audiovisual production, and a determination system to associate the message to a point, series 
of points, or one or more defined portions in the playback. In addition, the system includes a 
storage system for storing the message thread along with an indication of the associated point 
10 in the playback. Moreover, the system includes a displaying system for displaying the 

message thread adjacent to playback of the audiovisual production, and substantially at the 
associated point in time of the playback. 

[13] In yet another embodiment of the present invention, a computer-readable medium 
includes instructions for displaying message threads in connection with an audiovisual 

15 production. More specifically, the computer-readable medium includes one or more 

instructions for receiving a signal from a viewer interface to define alphanumeric text and or 
additional message content such as audio or images, and one or more, instructions for 
displaying the defined message adjacent to a first display of the audiovisual production. In 
addition, the computer-readable medium includes one or more instructions for displaying the 

20 defined alphanumeric text adjacent to a second display of the audiovisual production. 

[14] In yet another embodiment, the playback of a DVD at a master display system is 
synchronized with the playback of a DVD containing the same audiovisual production but 
which is resident in one or more slave DVD players. From the master computer, a viewer 
who initiates a group-viewing session is provided with DVD player commands to control 

25 playback at other computers or DVD players participating in the group-viewing session. This 
embodiment enables multiple viewers to exchange messages while viewing the same 
audiovisual production at the same time. To initiate a group viewing, a viewer at the master 
computer initiates a call to other viewers to join in and a list of viewers is created. From that 
point forward, the master computer controls the playback of the audiovisual production for all 

30 viewers. 

[15] To initiate a group viewing, a viewer at the master computer or other device initiates 
a call to other viewers to join in and a list of viewers is created. From that point forward, the 
master computer controls the playback of the audiovisual production for all viewers. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[16] Fig. 1 A illustrates presentation of an audiovisual production in conjunction with 
messages linked to time codes of the production. 
5 [17] Fig. IB illustrates presentation of alphanumeric text according to various 
embodiments of the present invention. 

[18] Fig. 2 is a block diagram of a method for electronic messaging synchronized to 
audiovisual production according to an embodiment of the present invention. 
[19] Fig. 3 is a block diagram of presentation of alphanumeric text according to an 
10 embodiment of the present invention. 

[20] Fig. 4 is a block diagram of presentation of alphanumeric text according to another 
embodiment of the present invention. 

[21] Fig. 5 is a block diagram of presentation of alphanumeric text according to yet 
another embodiment of the present invention, 
1 5 [22] Fig. 6 is a block diagram of presentation of alphanumeric text according to yet 
another embodiment of the present invention. 

[23] Fig. 7 is a block diagram of presentation of alphanumeric text according to yet 
another embodiment of the present invention. 

[24] Fig. 8 illustrates a viewer's sorting criterion according to yet another embodiment of 
20 the present invention. 

[25] Fig. 9 illustrates another viewer's sorting criterion according to yet another 
embodiment of the present invention. 

[26] Fig. 10 illustrates yet another viewer's sorting criterion according to yet another 
embodiment of the present invention. 
25 [27] Fig. 1 1 illustrates yet another viewer's sorting criterion according to yet another 
embodiment of the present invention. 

[28] Fig. 1 2 is a block diagram of a system for electronic messaging synchronized to 
audiovisual production according to an embodiment of the present invention. 
[29] Fig ? 13 is a block diagram of a system for electronic messaging synchronized to 
30 audiovisual production according to another embodiment of the present invention. 

[301 Fig. 14 illustrates a system configuration for implementing a master DVD player 
controlling a plurality of slave DVD players. 
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DETAILED DESCRIPTION OF THE INVENTION 
[31] In the following description of a preferred embodiment, reference is made to the 
accompanying drawings, which form a part hereof, in which is shown by way of illustration 
specific embodiment in which the invention may be practiced. In the following description, 

. 5 numerous specific details are set forth in order to provide a complete understanding of the 
present invention. It will be apparent to one skilled in the art that the present invention may 
be practiced without the specific details. In the development of the actual implementation, 
numerous implementation-specific decisions must be made to achieve the design goals that 
will vary for each implementation. Accordingly, in order not to obscure the present 

10 invention, well-known structures and techniques are not shown or discussed in detail. 
[32] According to the present invention, techniques for electronic messaging 
synchronized to media presentation are provided. In one embodiment, a display device for 
playback of an audiovisual production receives alphanumeric text or other message content 
entered or provided by viewers of the audiovisual production. One preferred flexible and 

1 5 fully featured web messaging system is available at http://scoop .kuro5hin.org/ (Scoop) which 
provides a robust engine for posting, classifying and gardening messages combined with a 
self-rating system. Another preferred messaging system is commonly referred to as "wiki" 
and implementation information may be found at http://www.wikipedia.com/wiki/wikipedia 
on the FAQ page. Essentially, a wiki refers to a web page that can be edited by the public 

20 using a simplified syntax. Users become a self-organizing community that gives greater 
weight to frequent users who tend to garden the page to provide more useful and relevant 
information. Other methods for generating and storing messages are known such as, by way 
of example, a chat room system, electronic mail, message board system or an instant mail 
system. The display device may be a computer system, a consumer digital device, other 

25 display, system, or platform, or combination thereof. A consumer digital device may be a 
video Compact Disk (CD) player, a DVD player, or other. 

[33] With reference to Fig. 1 A, the method of the present invention enables the viewer 
who desires minimal interactivity to passively experience the message stream or to 'turn-off 
the message portion of the display. However, any viewer can become more active in the 
30 messaging community at any time. The present invention creates a community of interested 
viewers by allowing an open messaging board that any owner of an enhanced DVD (that is a 
DVD that includes the message thread link) can post to. Viewers may create their own 
comments, respond to others' comments, and annotate the DVD video experience as they see 
fit - creating a compelling reason to view the DVD again and again, as well as creating brand 
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loyalty with both the subject of the DVD and the supplier of the DVD. Clearly, the present 
invention enables the performers, actors or the artists to create annotated "tours" of the 
content available on the DVD. The annotations guide the viewer through moments of video, 
giving an inside perspective on the event being shown. Viewers are able to navigate from 
5 one message event to the next, using "tour" navigation controls; or simply to watch the entire 
video and have the messages appear on-screen as the message events are triggered by the 
playback timeline. Alternatively, viewers may review messages and then jump to the 
location of the audiovisual production being discussed in the message. 
[34] The primary interface to the messaging application is initially unobtrusive; as such, 

10 the main interface (as well as the first interface a viewer will perceive when choosing to 

engage the messaging application) is a screen in which the video dominates the available real 
estate of the display as shown generally at 50. In this initial viewing mode, the headlines and 
any messaging control interface appears in a lower area of the screen in a manner that 
approximates an orientation which all viewers of modern television programs are familiar. 

15 More specifically, the headlines and viewing interface appear as a news-style view where the 
focus is always on the subject of the video and the textual and context information is 
displayed superimposed along a bottom portion of the screen as generally indicated at 60. 
[35] If the viewer does not wish to view the headlines, the messaging display may be 
deactivated until the viewer determines it desirable to view messages relating to the currently 

20 viewed portion of the audiovisual production. Accordingly, the viewer may switch between a 
full screen video display and full screen display of messages related to the current time code. 
In operation, when playback of the audiovisual production comes to an interesting portion, 
the viewer may select to switch to message display using the pause or a message button 
associated with the viewer interface. With the playback paused, the viewer has a full screen 

25 messaging window relating to the portion of the audiovisual production. The displayed 

. messages are linked to a time code most closely proximate to the playback time code. When 
finished reading the messages, the viewer returns to the playback of the audiovisual 
production at the time code where paused. During the time the messages are displayed, the 
viewer may select to display non-contiguous portions of the audiovisual production identified 

30 in one or more of the messages and then return to normal playback after reading the messages 
of interest. 

[36] When a DVD, having one or a plurality of audiovisual productions recorded 
thereon, is inserted into a DVD player, the DVD player identifies the DVD and contacts a 
remote messaging server to request a top-level index containing a list of audiovisual 
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productions that have messages on the messaging server. Typically, the messaging server is 
connected via the Internet or other digital communication network. 

[37] From the messaging server, the DVD player receives an index for message subjects 
associated to each audiovisual production on the DVD. Each subject index contains subject 
5 headings of messages directed to an audiovisual production, and the start and end time 

indications of each message subject. The start and end time indications may comprise start 
and end time codes associated with the audiovisual production or DVD index information, 
such as chapters. The DVD player may subsequently download the subject index for each 
audiovisual production, or alternatively the DVD player may utilize a predictive process and 
10 attempts to download only the subject index for the audiovisual production most likely to be 
viewed by the viewer. Alternatively, the DVD player may download the subjective index of 
an audiovisual production only upon selection by the viewer. 

[38] The downloaded subject index may be stored in the local disk cache of the DVD 
player or in a storage unit coupled to the DVD player. The cache memory may be temporary 

15 or preferably permanent. With permanent cache memory, once a subject index is 

downloaded it remains available for future uses. In future uses, the subject index previously 
downloaded onto the local disk cache may be compared to the current version of the subject 
index on the messaging server. The comparison determines whether any update to the 
subject index needs to be downloaded. Furthermore, when the DVD player downloads a 

20 subject index, it may also download body text of all messages related to the subject index. 
Alternatively, the DVD player may download body text of messages only when the start 
timecode of the messages has arrived, and consequently store the messages in the local disk 
cache awaiting viewer action. 

[39] Upon viewer selection, the DVD player or other display device plays back an 
25 audiovisual production, which may be a movie, a game, or other media presentation. The 

playback involves access to an audiovisual production stored on the display device, stored on 
a storage unit locally connected to the display device, or on a storage unit connected to the 
display device over a digital network. The digital network may be the Internet or other 
network. By way of example, a movie or a concert recorded on a DVD may be displayed by 
30 a DVD player or a computer system together with the messages obtained from the message 
server. The display screen may be a television screen, a computer screen, or a display area of 
other display device. , The display screen provides, in addition to windows for alphanumeric 
text and the audiovisual production, a viewer interface to receive the viewer's inputs, so that 
the viewer may resize the windows for alphanumeric text and audiovisual production. 
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During play back of the audiovisual production, the message is displayed as superimposed 
over a portion of the display of the audiovisual production. Alternatively, the message or 
alphanumeric text may be displayed in the lower third part of a display screen without 
overlapping the display of the audiovisual production. 
5 [40| As the audiovisual production plays back, text "headlines" (akin to the initial 
message subject headlines in standard threaded messaging systems) or visual icons (not 
shown) will appear - these headlines will be keyed to the time code of the video media. A 
viewer action - clicking on the headline or icon, or hitting the space bar on a keyboard or 
other user interface device - will change the messaging interface into the expanded message 
10 interface, which allows viewers to see the "body" of each message, as well as any replies to 
each message. If the viewer does not choose to view the expanded interface, headline 
comments will appear and disappear in the lower third of the screen in a manner that is well 
understood in the art. 

[41] An expanded message interface shrinks the display of the video to a smaller portion 

1 5 of the full screen, to allow space for the body text of each comment, as well as to provide a 
more advanced messaging system user interface controls. An "outline" view of the message 
headlines allows the viewer to see an overview of the available messages and the number of 
replies. The outline view follows the media timeline, and.has a simple scrolling interface, to. 
move through the messaging timeline. The body text area changes under viewer control to 

20 allow the user to read the comments in a leisurely fashion and retain primary focus on the 
. video media. In order to fit messages within the limited screen real estate and keep interface 
controls simple, there should be an arbitrary limit on how "deep" threads can go - it being 
recommended to limit the thread to a comment and one level of responses. 
[42] Refer now to Fig. IB where the various subjects of messages related to an 

25 audiovisual production are displayed in outline window 1 10. The audiovisual production 
may, by way of example, be a sporting event, a theatrical performance or video of a music 
concert. As the audiovisual production plays back, the subject of initial messages in a 
threaded messaging system, appears on the display screen. If one of the message threads 
appears of interest to the viewer, the viewer may select a message thread by clicking on the 

30 subject headline, hitting the space bar on the keyboard, or other viewer action. Once 

selected, the display window for the message expands to present the body text or image of the 
messages associated with the selected subject. The displayed body text or image in the 
expanded window changes in accordance with the viewer's further directions. The expanded 
window also provides a viewer interface for advanced control of the messaging system. If 
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the viewer does not select any message subject, the subject headline appears and disappears 
periodically in the lower third part of the display screen. An outline window 1 1 0 is located to 
the right of a display window 120 where the audiovisual production is displayed. Outline 
window 1 10 may be placed in other locations on the screen depending on the needs for a 
5 particular application. Outline window 1 10 presents subjects of initial messages and number 
of replies. The message subject at the bottom of the outline window 110 may have a start 
time code that substantially corresponds to the timeline point of the audiovisual production 
that the viewer is viewing. In addition, outline window 110 also provides a simple scrolling 
interface for the viewer to move through various subjects related to the audiovisual 

10 production. As indicated in Fig. IB, the darkened highlight indicates that the message thread 
relating to the subject "Benny D: King's finest kicks" is selected. Upon selection, the 
timeline of the audiovisual production is changed to the start timecode of the subject "Benny 
D: King's finest kicks." Consequently, the playback of the audiovisual production jumps to 
the frame corresponding to the start timecode of the selected subject. The start timecode of 

15 the subject "Benny D: King's finest kicks " is at about 01 :23: 12:18 as shown in timeline 
clock 130 and in timeline indicator 140. Upon selection of the subject "Benny D: King's 
finest kicks," the first line of the initial message under the selected subject is shown in 
headline window 150. In addition, body text of the initial message and responses to this 
message are shown in expanded message window 160. 

20 [43] For another example, the maker of an audiovisual production may create an artist- 
annotated "tour" of a concert. The "tour" comprises various subjects, each of which 
corresponds to an event of the audiovisual production. Under each subject, the annotator 
would provide the viewer with an inside perspective on the corresponding event of the 
audiovisual production. The viewer may jump from one event to the next event of the 

25 audiovisual production by selecting various message subjects, substantially as described 
above. 

[44] According to another embodiment of the present invention, as shown in Fig. 1 A, the 
subject "Benny D: King's finest kicks " has a start timecode at about 0 1 :23: 1 2: 1 8 of the 
timeline of the audiovisual production, as indicated in timeline clock 1 30 and timeline 
30 indicator 140. Therefore, when the playback of the audiovisual production reaches point 
01:23:12:18 of its timeline, the subject "Benny D: King's finest kicks " is automatically 
displayed at the bottom of outline window 1 10. The first line of the initial message under this 
subject is presented in headline window 150, and body text of messages related to this subject 
are displayed in expanded message window 160. Further, the playback of the audiovisual 



production may be controlled through viewer interface 170 with various functions such as 
fast forward, fast rewind, play, pause, and others. For example, the maker of an audiovisual 
production may create an artist-annotated "tour." With the playback of the audiovisual 
production, the annotator may guide the viewer through various points of timelineand 
5 provide an inside perspective on the event being shown. 

[45] According to yet another embodiment of the present invention, as shown in Fig. IB, 
the viewer may select to input his own comments under a new subject or in response to an old 
message. Upon selection, expanded message window 160 for displaying alphanumeric text 
or other message content instantaneously displays the message as input by the viewer. 

10 Further, outline window 1 10, at its bottom, displays the message subject whose start timecode 
substantially corresponds to the point of timeline the viewer is currently viewing. In addition, 
the first line of the initial message under this subject is presented in headline window 150. 
For example, at about 01:23:12:18 as shown in timeline clock 130 and timeline indicator 140, 
the subject "Benny D: King's finest kicks" is displayed at the bottom of outline window 1 10. 

1 5 The first line of the initial message under this subject "Benny D: King's finest kicks" is 
presented in headline window 1 50. 

[46] Fig. 2 is a block diagram for electronic messaging synchronized to media 
presentation according to an embodiment of the present invention. The method includes 
three processes: receipt of alphanumeric text 210, playback of audiovisual production 220, 

20 and presentation of alphanumeric text 230. As used herein, the phrase alphanumeric text 

includes messages that may include or other message content such as media presentation, still 
images or voice files. At process 210 of receipt of alphanumeric text, a display device for 
playback of an audiovisual production receives alphanumeric text entered by viewers of the 
audiovisual production. At process 220 of playback of audiovisual production, an 

25 audiovisual production is displayed. At process 230 of presentation of alphanumeric text, 
alphanumeric text of a message is displayed when playback of the audiovisual production is 
performed. 

[47] According to one embodiment of the present invention, as shown in Fig. 3, process 
230 of presentation of alphanumeric text comprises three sub-processes: process 310 for 
30 displaying alphanumeric text, process 320 for accepting viewer's input, and process 330 for 
resizing display window for alphanumeric text, display window 120 for audiovisual 
production, or both. 

[48] According to another embodiment of the present invention, as shown in Fig. 4, 
process 230 of presentation of alphanumeric text comprises four sub-processes: process 410 
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of displaying subject headers of messages, process 420 of selecting a subject header, process 
430 of playback of segment of audiovisual production, and process 440 of displaying body 
text of messages. 

[49] According to yet another embodiment of the present invention, as shown in Fig. 5, 
5 process 230 of presentation of alphanumeric text comprises two sub-processes: process 510 
for displaying a message subject whose start timecode substantially corresponds to the point 
of timeline the viewer is currently viewing, and process 520 for displaying body text of 
messages under this subject. 

[50] According to yet another embodiment of the present invention, as shown in Fig. 6, 
1 0 process 230 of presentation of alphanumeric text comprises two sub-processes: process 610 
for accepting viewer's input and process 620 for displaying viewer-inputted message. 
[51] Another embodiment of the present invention is shown in Fig. 7. A process 230 of 
presentation of alphanumeric text comprises four sub-processes: process 710 of accepting 
viewer's criterion for sorting messages related to an audiovisual production, process 720 of 
15 accepting viewer's criterion for displaying messages, process 730 of sorting messages, and 
process 740 of displaying messages according to the sorting result. These sub-processes may 
be implemented in conjunction with various processes described in various embodiments, as 
shown in Figs. 1-6. 

[52] When a user inserts an encoded DVD, computer code recognizes the encoded 
20 information and automatically contacts a messaging server. The DVD display platform (such 
as a computer system having a DVD player and a display) requests a top-level index 
consisting of a list of media files on that DVD that have messages, as well as the location of 
the subject headline index for each piece of media recorded on the DVD. A predictive 
process attempts to pull down the headline index for the "most likely" first-viewed piece of 
25 media, which contains the subject headings along with the start and end time codes for each 
piece of media, so that the response time of the messaging application will appear to the 
viewer as more or less instantaneous. As the viewer selects to view any particular segment of 
the audiovisual production, a headline/time code index for each production is downloaded 
and stored to a local disk cache associated with the DVD platform. Preferably, the cache is 
30 persistent so that once an index has been downloaded it is available for future use. The cache 
can subsequently be compared to the message server indexes to determine if any updates 
have occurred since the last play of the disc. Another predictive process will be employed to 
"pre-download" upcoming body text, so that the entirety of each message is available 
precisely at the start time of the event. As the starting time codes of message events continue 
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to pass, the application will download body text, and store it in the local disk cache awaiting 
user action. As ending time codes pass, the headlines will be removed from the main user 
view, though message headlines will always be available for perusal through a simple 
back/forward interface independent of media rewind/fast-forward. 
5 [53] The present invention tracks the playback time code at all times, and follows as 
closely as possible a viewer's interruption of the normal playback timeline. For example, if a 
user fast-forwards or rewinds normal playback to a new time code, the headline display 
window will show the headlines appropriate for that time code. When a viewer clicks the 
action item of any headline, the body display window will show the full text of that message. 

10 The display of the body of any message will stay persistent until user action replaces it with a 
new message in order to allow the user to read the full text of the message. The message body 
. section has a simple back/forward interface independent of media playback, similar to the 
headlines, and it further includes the control mechanism to reset the media playback to the 
start and/or end of the displayed message event. Thus, the viewer may jump to a particular 

1 5 time code after they have had the time to read the text of the message. 

[54] At times, the viewer may desire to sort messages and selectively filter out some of 
these messages. As shown in Fig. 7, at process 710, the display device receives viewer's 
sorting criterion. The sorting criterion may be by message ages as shown in Fig. 8, age of '.. 
messages under the same subject as shown in Fig. . 9, ratings of messages as shown in Fig. 10, 

20 ratings of messages under the same subject as shown in Fig. 11, ratings of authors, or other 

criterion or combination thereof Ratings of messages reflect feedbacks of readers on quality . 
of messages. Ratings of authors reflect cumulative ratings of messages sent by particular 
authors. Viewers may enter their rating by selecting a Rate It button such as shown at 1 120 
in Fig. 1 1. 

25 [55] At process 720, the display device receives the viewer's display criterion, such as 
displaying only messages sent by a particular author, displaying only messages with ratings 
higher than a threshold, or displaying only messages sent by authors whose ratings higher 
than a threshold. At process 730, the display device sorts messages according to the sorting 
criterion as described above. At process 740, the display device displays only messages that 

30 satisfy the display criterion. 

[56] Rating of messages and rating of authors as discussed above gains importance as the 
present invention grows in popularity. The amount of messages may grow to a point where it 
would not be possible to properly experience each and every message event in any given 
subject category. The viewer would need to be able to selectively filter out the messages. 
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For example, the viewer may want to follow messages only by a particular author, e.g., a 
friend or an artist associated with the audiovisual production, or messages under a particular 
thread. In addition, the viewer may need to not only rate fellow viewers as to quality of their 
postings but also filter out messages based on ratings. A threshold system that allows the 
5 viewer to select a level below which messages are filtered from the interface is 

recommended. As the viewer posts messages, other viewers or a system administrator can 
rate the quality of these messages, and the ratings may provide a cumulative score for each 
viewer. Viewers of the messages would then be able to select a score below which they do 
not wish to view, and thereby effectively filter out garbage or irrelevant comments. In 
10 addition, the artist and/or a record label may give a viewer additional points to his or her 

rating, and thus raise the viewer to prominence in the community. The record label may also 
bless a particular author, e.g., an artist associated with the audiovisual production, so that his 
or her messages are always available for viewing regardless of the viewer's threshold filter 
setting. 

1 5 [57] Fig. 1 2 is a block diagram for a system 1200 for electronic messaging synchronized 
to presentation of the audiovisual production, according to an embodiment of the present 
invention. System 1200 comprises four sub-systems: receiving system 1210, determination 
system 1220, storage system 1230, and display system 1240. System 1200 can perform 
various methods of electronic messaging synchronized to media presentation as described 

20 above in Figs. 1-11. Receiving system 1210 receives inputs from a viewer interface to define 
display of a message containing alphanumeric text, still photos or a voice message file during 
playback of an audiovisual production. The inputs may include viewer's command for 
resizing display window for alphanumeric text or other messages, viewer's command for 
resizing display window for audiovisual production, viewer's command for selecting a 

25 particular message subject, viewer's input of comments on an audiovisual production, 
viewer's input of criterion for sorting messages, viewer's input of criterion for displaying 
messages, or other viewer input. 

[58] Determination system 1220 associates a message with a point of timeline in 
playback of an audiovisual production. For example, a message may associate with the 
30 audiovisual production according to the start timecode of the message, the end timecode of 
the message, or other characteristic of the message. 

[59] Storage system 1230 stores the message along with its association with a point of 
timeline for an audiovisual production. The storage may be performed on a storage device 
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remotely located from the display device of the audiovisual production, on a local disk cache 
of the display device, on other device, or combination thereof. 
[60] Display system 1240 displays a message during playback of the audiovisual 
production. The display of the message may occur at its associated point of timeline of the 
5 audiovisual production or at any time independent of playback of the audiovisual production. 
[61] In another embodiment, each message includes an embedded icon. The icon is 
linked to a specific location in the playback of the audiovisual production. When the icon is 
selected, playback jumps to specific location of the audiovisual production that the icon refers 
to and initiates playback of a defined portion of the audiovisual production. For example, 

10 beginning and ending points can be specified for the defined portion. More specifically, 

when a viewer is reading a message, it is possible to select the icon and immediately jump to 
the corresponding location in the audiovisual production. Playback proceeds by playing a 
selected portion, or a clip, of the audiovisual production. Often, a viewer will explore an 
audiovisual production by first reading messages and then pre- viewing portions of the 

15 production by clicking on one or more icons in the messages to pre-view specific portions. 
To illustrate this feature, a viewer may read messages that critique an audiovisual production 
and then jump to view several defined clips identified by the message before deciding 
whether to initiate the playback of the entire audiovisual production. 
[62] The message icon represents a specific location on a specific DVD that can be 

20 specified by the author of the message. In one preferred embodiment, a software tool 
executing on a computer associated with the messaging device enables the creation of a 
message icon and establishing the link to a specific portion of the audiovisual production. 
The messaging device may be a computer system, cell phone, PDA or other computer-based 
viewer interface system. 

25 [63] According to yet another embodiment of the present invention, Fig. 1 3 is a block 
diagram for a system 1300 for electronic messaging synchronized to media presentation. 
System 1300 comprises messaging server 1310, viewer interface 1320, and administrative 
interface 1330. System 1300 may include some or all four subsystems of system 1200. 
[64] Messaging server 1310 preferably includes a standard database-driven threaded 

30 messaging system, enhanced with an identification of the DVD audiovisual production (DVD 
ID) and the start/end time codes for each message. The use of a standard servers and 
database affords great flexibility in delivering future functionality, and allows traditional, . 
browser-based interaction with the database of messages. In this manner, message threads 
are readily integrated with streaming media versions of the audiovisual production that allows 
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a DVD-like experience of the time-based messaging system on the Internet. The messaging 
database is preferably separate from any pre-existing user database, in order to foster 
modularity in implementation. By using existing open-source messaging server code, a 
consistent application program interface (API) is provided together with a framework for 
5 future feature enhancements while preserving backwards compatibility. Open source software 
also allows for easy customization, so integration of additional time code/DVD ID tags that 
are necessary to allow the messaging system to interact with third party media content. 
[65] In addition to the DVD ID and start/end time code tags (or other indicators) that 
need to be present for each thread, the present invention contemplates the use of a user 

10 "karma points" system in which each contributing viewer is assigned a rating. The rating 

functionality only needs to be employed where the messaging system is open to end-users. If 
the DVDs that employ the messaging system are limited to officially created artist "tours" 
1 there is no need to make use of the rating system functionality. In one preferred embodiment, 
a user's rating is determined by the community of viewers where every message allows 

1 5 viewers to rate the quality of its content, and every message author will accumulate a good or 
bad rating based on the judgment of the community. 

[66] Messaging server 1310 may employ various types of messaging systems such as 
instant messaging system, chat room systems, email systems, message board systems, 
standard database-driven threaded messaging system enhanced with DVD ID and start and 

20 end time codes for each message, or other messaging system. The standard messaging 

system provides the viewer with browser-based interaction with the message database and 
facilitates integration of the message database with streaming media versions of audiovisual 
productions. This integration provides the viewer with playback of an audiovisual production 
that is downloaded over the Internet and display of messages associated with the audiovisual 

25 production. In addition, the standard messaging system also provides flexibility in delivering 
additional functionality. 

[67] Viewer interface 1320 may provide various display windows configurations upon 
viewer's selection. The display windows include those for audiovisual production, for body 
text of downloaded messages, for body text input by the viewer, for various message 
30 subjects, for message filter, for viewer login, and other information and control display. 
[68] After a viewer logs in viewer interface 1320, viewer interface 1320 first presents 
default display windows. The default display windows comprise one for audiovisual 
production and one for message subjects associated with the audiovisual production. When 
the audiovisual production plays back, the message subjects associated with the audiovisual 
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production are also be displayed. The message subject at the bottom of the list usually 
substantially corresponds to the point of timeline that the viewer is viewing. Further, viewer 
interface 1320 provides means for the viewer to scroll up and down the subject listing. 
Clicking on any message subject brings body text of messages under this subject into the 
5 expanded message window. At substantially the same time, playback of the audiovisual 
production jumps to the point of timeline corresponding to the selected subject. 
[69] On viewer interface 1320, the display window for the audiovisual production may 
take up most or other percentage of the available screen real estate. Icons for controlling 
playback of audiovisual production may be displayed in the display window. Alternatively, 

1 0 the stream of messages and control icons may appear in the lower third part of the screen real 
estate. The control icons allow the viewer to view body text of messages and to enter 
viewer's comments in expanded message window for alphanumeric text. The expanded 
message window may provide only two or three lines of body text of each of the messages in 
a large and readable font. A slow "auto scroll" effect may be employed should any body text 

1 5 extend beyond the viewable range of the expanded window in order to minimize the need for 
viewer interaction when the viewer is viewing the audiovisual production. The window for 
displaying body text and the window for entering the viewer's own comments may be similar. . 
For both types of windows, authors and headlines of messages may be presented along with 
body text of the messages with a slow "auto-scroll" effect. When the display window for 

20 entering viewer's own comments is selected, the viewer interface may also provide controls 
for the viewer to associate the composed message to a specific point of timeline for the 
audiovisual production. In addition, on viewer interface 1320, controls for replying to 
displayed messages, for rating the messages, and for drafting a message under a new subject 
may also be displayed. 

25 [70] In addition, a timeline indicator, a timeline clock, or both may appear below the 

display window on viewer interface 1320 for audiovisual production. The timeline indicator 
140 and the timeline clock 130 (see Fig. 1) show the point of the timeline at which the 
audiovisual production is currently being displayed. Clicking on the timeline indicator 
allows the viewer to navigate to points of the production more easily than using the standard 

30 rewind and fast-forward controls. 

[71] An administrative interface is also provided to raise or lower a given user's rating if 
the artist and/or label so desires. Administrative interface 1330 may provide several sub- 
interfaces that perform various administrative functions. These sub-interfaces comprise 
subject category selection interface, administrative filtering interface, administrative login 
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interface, and administrative rating interface. Subject category selection interface provides a 
simple method for selecting subjects of messages the viewer desires to follow. 
Administrative filtering interface provides a simple method for selecting a rating "threshold." 
Any messages with a rating below this threshold would not be displayed. In addition, 
5 administrative filter interface may also provide a simple method for selecting any particular 
author whose messages the viewer desires to follow. Administrative login interface displays 
viewer name and password entry fields. When a viewer logs in to the system for the first 
time, the viewer may select a "screen name" or an alias, by which the viewer is known to 
other viewers in the system. 

10 [72] The rating system will discourage "graffiti'Mike postings where the poster has 

nothing appropriate to say, and encourage posts that are thoughtful and relevant to the topic. 
The user rating may also be employed to control access to posting messages - i.e. a user who 
is rated poorly may be blocked from posting further messages, or their access to post 
messages in certain forums might be curtailed. In addition, the user level can be employed to 

15 filter out display in the main video window of the interface when the number of messages in 
the system increases and there will be too many to display and perceive properly. An 
algorithm operating on the messaging server gives priority to comments from users that have 
higher ratings - again encouraging quality posts. Administrative rating interface allows 
raising or lowering a viewer's rating. The viewer's rating may be employed to control access 

20 to posting messages. For example, a viewer with a poor rating may be blocked from posting 
messages, or their access to posting messages in certain forums may be curtailed. 
[73] In addition, the start timecode and end timecode for some messages may be only a 
few seconds apart. Therefore, the viewer may not fully understand body text of these 
messages before they are replaced by subsequent messages. Consequently, according to yet 

25 another embodiment of the present invention, the system would set a minimum time that each 
message must appear on screen. If the system determines that within the minimum time, the 
messages being displayed would overlap with subsequent messages, the messages possessing 
certain characteristic, such as being composed by authors with high ratings, are displayed. 
[74] Moreover, when a message subject draws strong interest from viewers of the 

30 system, this message subject creates a "hotspot" in the timeline of the audiovisual production. 
At a hotspot, a large number of messages aggregate within a short period of the timeline. 
Consequently, displaying all of these messages in the limited screen real estate may be 
difficult. Hence, the number of displayed messages would preferably be limited. For 
example, only messages composed by highest rated authors are displayed, even if an 
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individual viewer's rating threshold is set lower. In contrast, when messages at a point of 
timeline are sparse, messages by composers with low ratings may appear, so long as the 
composers' ratings exceeds the viewer's rating threshold. 

[75] Communication between the messaging server 13 1 0 the Administrative Interface 
5 1 330 and the Viewer Interface 1320 take place preferably within an encrypted channel that 
may also be used to send tour calendar, viewer demographic, and usage profile information. 
An intermittent connection between the message server and database and the DVD platform 
is attempted every time a DVD is launched. The connection stays persistent only as long as 
is required to register and upload user/demographic data. However, a more persistent 

1 0 connection to the database is required so that the messages entered by users can be updated in 
a fashion approaching real-time chat. Also, as the message boards grow larger, the amount of 
data transfer required to download an entire message board may become impracticable over 
typical modem connections. However, a real-time, persistent connection presents challenges 
in application design, as well as scalability so the preferred method is that a hybrid of one- 

15 time download and persistent connections will be employed. Message board subject indexes 
and headline titles are downloaded at launch when a user logs into the system - and then the 
full text of messages can be downloaded as necessary. 

[76] A program code may be implemented to instruct a computer system, a consumer 
digital device, a central processor-based device, a custom circuitry or other device to perform 
20 various processes of present invention as discussed in Figs. 1-11, including receipt of 

alphanumeric text, playback of audiovisual production and presentation of alphanumeric text. 
In addition, the computer code may also instruct a device or a system to perform various 
functions as described with respect to Figs. 12 and 13. 

[77] Another embodiment of the present invention is shown in Figure 14 where a master 
25 computer coupled to a first DVD player 1402 is further coupled by a communication network 
1404 to one or more other DVD players, such as represented by stand-alone DVD player 
1406, MPEG player 1408 and DVD player 1410. Communication network 1404 is 
preferably the Internet or other worldwide communication network that enables digital 
transmission of audiovisual productions from one communication device to many 
30 communication devices. In this embodiment, the playback of a DVD at a master display 
system, comprising by way of example DVD player 1402 and computer system 1412, is 
synchronized with the playback of the same audiovisual production stored on DVDs at slave , 
playback devices such as DVD players 1406 - 1410. Although players 1406 - 1410 are 
denominated as slave players, it will be apparent to one skilled in the art that any one of such 
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players may at some subsequent time be designated as the master player and player 1402 
designated as a slave player. 

[781 The player having the designation as 'master' and the control of all slave players 
may be transferred from the current master to any one of the slave players by passing a token 
5 1414 from the master to the slave. If more than one slave has an outstanding request for 
token 1414, the token may be passed in a round robin fashion. During other audiovisual 
presentations, two or more DVD players may exchange the token in a 'ping-pong' fashion. 
This type of token transfer may be particularly useful in a panel discussion where different 
portions of the audiovisual presentation is germane to different panelists. Regardless of the 
10 manner used to pass the token, whoever has the token may decide to retain the token for the 
entire duration of the audiovisual performance in which case the other DVD players remain 
in slave mode. 

[79] As illustrated in Fig. 14, each of the DVD players may be of various types and 

designs. For example, players 1402 and 1408 are coupled to computer systems 1412 and 

15 1414, respectively, each of which includes a user interface and a viewing device such as an 
LCD display. Other DVD players, such as digital recording and player device 1406, such as 
is commercial available under the brand name "TiVo," may be a recording and. playing 
device having controls on the front face of the device and the audiovisual production is 
displayed on an associated digital TV such as found in most home theater systems 1416. One 

20 requirement for standalone DVD players is that the device provides an Internet connection. 
Still other DVD players may be a MPEG player 1408 examples of which include 
commercially available game consoles capable of playing a DVD and connecting to the 
Internet such as the Sony PlayStation®, Microsoft X-Box® or Nintendo GameBoy®. MPEG 
player 1408 is typically coupled to a display device 14 such as a television although player 

25 1408 further may include a small LCD display 1418. 

[80] The system illustrated in Fig. 14 further includes a messaging server 1420 that 
provides a chat room for viewers wishing to participate in a presentation. To initiate a group 
presentation, an initiator solicits participants by identifying the audiovisual work to be 
discussed and the time for initiating the presentation. Other viewers may join the 

30 presentation by indicating their desire to participate. Messaging server 1420 creates a list of 
viewers and their respective IP addresses. Each viewer is then advised to place a DVD disk 
containing the audiovisual presentation into his or her DVD player. If appropriate control 
software is not present on the computer associated with each DVD, server 1420 provides the 
software so that each DVD player may be controlled in a slave mode. 
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[81] At the start of the presentation, the master DVD player 1402 checks for the presence 
of the audiovisual presentation at each slave DVD player and initiates the playback of the 
master DVD and each of the slave DVD players. During the playback, each viewer may 
submit electronic messages to the moderator or moderators. If the message requires the 
5 moderator to change the normal sequence of the playback, the appropriate controls are 
entered at the master DVD player and transmitted via the communication network 1404 to 
each of the slave DVD. To illustrate, if a viewer at DVD player 1408 asks for clarification of 
a point in the audiovisual presentation, that message may be sent to the moderator at master 
system 1412 and each of the other viewers on their displays. If the moderator decides that it 

10 is desirable to pause the audiovisual presentation, a pause command is sent to the master 
DVD as well as each of the slave DVD players. Similarly, so long as the moderator retains 
the token designating his or her system as the master system, the moderator retains control 
over each slave DVD. Thus, the moderator can initiate play, pause, jump to a specific time 
code of the audiovisual presentation in a non-sequential manner at each slave computer. 

15 [82] During playback of the audiovisual production, server 1420 associates messages 
stored in its local memory with a corresponding time code. These messages may be 
communicated to the master and each slave display upon detection of the corresponding time 
code during playback of the audiovisual production. 

[83] Since server 1420 manages the participation of each slave DVD, a viewer may 
20 disconnect from the presentation at any time and the master DVD will no longer control his 
or her DVD. This embodiment enables on-line collaboration relating to an audiovisual 
production between two or many viewers. Advantageously, many viewers can chat live 
about the content of an audiovisual production because viewers are watching it together. 
[84] If there are two or more moderators, server 1420 may receive requests from the 
25 moderators for the token. The token may be passed by updating the viewer list maintained by 
server 1420 such that only the new master DVD may control the slave DVDs. This approach 
is especially useful where people are collaborating online about video edits to an audiovisual 
production when working from a rough edit on DVD or where DVDs are distributed to many 
viewers and the DVD is the passkey that authorizes the viewer to participate in the 
30 presentation. 

[85] With device 1406, a viewer may record a television show, sporting event or movie 
when aired on cable or television channels. At a subsequent point in time when it is 
convenient for the viewer to view the recorded show, event or movie, a communication link 
is established with message server 1420 and message links are downloaded and attached to 
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the recorded show, event or movie for viewing. Further, instead of using a DVD or computer 
disk for media storage, the audiovisual production may be transferred to a display device 
from a remote source, such as a media server computer, by way of a cable or satellite 
communication system. 

5 [86] Although the invention has been described primarily with respect to playback of 
productions from a DVD, any storage media or playback technology, such as Video-On- 
Demand cable or satellite systems can be used as the source of the audiovisual production. 
Further, the DVD playback device and the messaging device can be separate devices. For 
example, a cell phone or personal digital assistant (commonly referred to as a PDA) can be 
1 0 used as a messaging and control device while the audiovisual production is playing on a 
remote DVD player connected to the Internet. 

[87] Also, although a client-server architecture has been described, other network 
arrangements such as peer-to-peer can be used. In general, any type of network and 
arrangement of processors, e.g., computers, consumer electronic devices, central processor- 

1 5 based devices, and custom circuitry, can be used for transmission of data, playback, 

execution of a viewer interface and to perform any of the functions and features described 
herein. Processing can be performed at any physical location and can be done by a single 
processing unit, or in a co-processing or parallel arrangement with multiple processors. 
Further, although time codes for messages have been described, any other types of time 

20 indications that associate the messages to points in time of an audiovisual production may be 
used. 

[88] While certain exemplary preferred embodiments have been described and shown in 
the accompanying drawings, it is to be understood that such embodiments are merely 
illustrative of and not restrictive on the broad invention. Further, it is to be understood that 
25 this invention shall not be limited to the specific construction and arrangements shown and 
described since various modifications or changes may occur to those of ordinary skill in the 
art without departing from the spirit and scope of the invention as claimed. Thus, the scope 
of the invention is to be determined solely by the appended claims. 
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